DOCUMENT RESUME 



SB 235 204 



TM 830 617 



AUTHOR 
TITLE 

PUB' DATE 
NOTE 



PUB TYPE 



Koch , William R. . .. 

The Analysis of _ Dichotomous Test Data Using Nonmetric 
Mult idimens i onal Scaling. 

Apr 83 . 

25p. ; Paper presented _at the. Annual Meeting of the 
American Educational Research Association (67th, 
Montreal, Quebec, April 11-15 , 1983 ) . 
Speeches/Conference Papers (150) — Reports - 
Research/Techn ical ( 14 3 ) 



EDRS PRICE 
DESCRIPTORS 



I DENT IF J ERS 



MF01/PC01 Plus Postage. 

Achievement Tests; *Cluster Analysis; ^Latent Trait 
Theory; *Mul t i d imen s i onal Scaling; Multiple Choite 
Tests; *Re search Methodology; *Test I terns 
*Facet Analysis 



ABSTRACT ___ ; : : - - 

The techn ique of nonmetr i c mult idimens i onal scaling 
(MPS) was applied to real item response data obtained from a_ 
multiple-choice achievement test of unknown dimensionality. The goal 
was to classify the^50 items into the various subtests from which 
they were drawn originally, the / latter being unknown to the 
investigator. Issues- addressed in the research included 
dimensionality,, choice of item proximity measures, and 
appropriateness of the MDS model for analyzing dichotomous item 
response data. Three coefficients were chosen to form proximity 
matrices that ref lecteds^the associations of each item with each of 
the other items. These matrices then served as input to both the 
ALSCAL and MINISSA computer programs for MDS analysis. A three 
dimensional solution was found to be optimal, based on stress, the 
multiple correlation coefficient, and coef f icient of alienation 
values . Both cluster analysis and regional ( facet theory )_ analysis 
the spatial configuration were used to interpret 
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UTS 

Introduction 

£-3 . The problem of determining the "underlying structure" of a. set of 

LU interrelated variables, in some data reduction sense, i s. extremely diffi- 

cult, quite subjective, and has no unique solution. Thi s formidable task 
: becomes compl i cated : even further when the variables of interest happen to 

be scored di chotomously . However just such a situation confronts the 
researcher when analyzi ng data collected from a sample of examinee! 
responding to multiple-choice, achievement test items. 



s In the past, typical reasons for attempting to determine the dimension- 
ality of a test have included the desires to obtain scores for persons on 
each of- its separate subscal es or to provide evidence for the construct 
validity of the test. More recently, another moti ve. has been the desire 
to form 'unidimensiQnal subsets of items so 'that each sublet may be sub- 
jected separately to latent trait item calibration (parameter 
estimation). The -latter procedure is necessary to meet the basic assump- 
tion of item Response models that only a single trait is bei ng measured by 
the set of items that comprise a test. 



Traditionally, factor analysis has been the preferred method for 
attempting to determine the underlying structure of a test. However, 
artificially dichotomized item response data present some problems for 
the linear factor -analyti c model . For example, the model assumes that the 
variables have continuous distributions and that the variables have line- 
ar relationships both with each other and with the derived factors. 
Furthermore, the coefficients used to form the input item intercorre- 
iation matrix are typically either pii coefficients or tetrachoric 
coefficients. The former lead to arti factual item difficulty factors 
(McDonald & Ahlawat, 1974), while, the latter present other problems such 
as non-Grami an matri ces "(Chri stof f erson^; 1975). 

Another complication is that examinees have some probability of cor- 
rectly answering multiple-choice test i terns/ simply by chance , with the ' 
incidence of guessing increasing for very difficult items and/or for exam- 
inees with low achievement levels. As a result, guessing is usually con- 
sidered to be a nonlinear function of achievement level, a situation that 
cannot easily be accommodated by the linear factor analytic model. Final- 
ly, the results of factor analysi s a^e presented in terms of composite 
linear vectors (factors) at various angles to each other in space whictv^ 
are labeled subjectively for interpretation. 
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Compared to factor analysis, the _ assumptions about the input data made 
by nonmetrlc multidimensional scaling (MDS) are much less strict.- MDS 
requires only that order information be present in the data and that the 
variables be. related to each other at least ^monotonical ly. A large varfef 
ty.of coefficients is. available to measure the proximities/ or ..associ- 
ations,, of each variable with every, other variable, the onl-y requirement 
being. that the values, of the proximities may be ordered' in magnitude. No 
distributional assumptions are made about the variables. 

The goal of the MDS algorithm 1 s to approximate a one-to-one monotonia 
relationship betweeh the ordinal proximity information extracted from the 
input data and the' rankings of the corresponding distances among the items . 
located as points in the f i na_i_- spatial configuration . .Specifically, the 
rule to be met is the following: _jf the si mi 1 ari ty between stimulus i and 
stimulus j is greater than the similarity between stimulus k and stimulus 
1, then the distance between points i and j in.the solution space should 
be less than or equal to the distance betweenv : poi nts k.and.l: In the case 
of mul-tiple-chbice data, the stimuli are items and the items are posi- 
tioned as points in the solution £pace. 

?. _ — _ .* 

Similar to factor analysis, the MDS solution is represented using a set 
of linear dimensions... Howeyer, the dimensions serve simply as arbitrary 
coordinate axes to locate, "the variables, in space and usual ly • have" no 
intrinsic meani ng in and of . themsel ve%. Rather, several methods may .be 
used for interpretation of the results, such as discovering meani ngful 
regions of the. space, identifying clusters of the' vari abl es , and" finding 
directions (linear or nonlinear) of stimulus variation in the space. 
These techniques Will be discussed later in the paper. 

Unfortunately , MDS methodol ogy has its own set of < poten t.i al problems 
when appl ied to : dichotomous item response data. Firs_t,_the dat^a in. their" 
original form are: arranged in a rectangular array.of 0,1 responses from n 
examinees to • k A 'i terc.s . In order to be^appropriate for classical MDS input, 
these data must be transformed into a* ( k x k) proximity matrix by means of , 
some chosen coefficient of _ associ ati on . Also, in order to minimize the 
effect of item* difficulty levels on the dimensionality of the solution, 
the .coefficient used must be relatively unaffected by. vari at ion s irvi tern 
difficulty, Furthermore, to the extent that guessing on items introduces 
substantial random error into the data, the fit of the MDS model to the 
data is jeopardized. Despite these probl ems , one would 'still expect 
dichotomous item response data , properly transformed , to meet the minimal 
assumptions of the MDS model. 

However, MDS has rarely been applied in the context of analyzing dicho- 
tomous item response data > . Consequently , very, little is known about the 
properties or value of MDS methodology in such. situations . The present 
research was intended to contribute to the knowledge base and experience 
in' thi s area . 

Dimensional ity 

^ 

The novel task confronting the i nvesti gators *i n i thi s symposium was to ^ 
undertake a "bl i nd" analysi s of some real test '"data in an attempt to " 
determine .its underlying dimensionality, or structure. The moderator 
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constructed a matrix of item response data from a sample of 2,794 exami- 
nees to 58 assorted items. The Items were, chosen del iberately from 
diverse subtests and aggregated to form. a realistic, inherently multidi- 
mensional test. The moti vati.or, s behind these . procedures Were to seek a 
fair, rigorous challenge for the analytic techniques being compared and to 
provide a known criterion again:-;, which solutions could be evaluated. The 
analyses were "blind" in the sens -i that only the moderator and the discus- 
sant knew the actual source and . l intent of: the items and, therefore, the 
"true" dimensionality of the test. ; : 

Dimensional i ty is a particularly slippery concept, and it becomes a 
technical term only in specific contexts. For example, in the variance 
decomposition framework of factor aralysis, it is common to think of the 
derived factors as the dimensions oi the multivariate space. This notion 
is reasonable in* the sense that the retained linear composi te variables 
(factors) are the number of orthogonal • dimensions necessary to represent 
adequately or to "account for" the common variance present in the original 
set of variables. 

In the context of MDS, dimensionality is frequently thought of as bei-ng 
the minimum number of orthogonal coordinate axes necessary to accommodate 
the order relations present among the input/variables by means of their 
interpoint distances in Euclidean space. Others refer to dimensionality 
when interpreting and labeling directions through the solution space. 

Finally, in'the context of latent trait theory, Lord (1980) suggests an 
empirical definition, of un i di mpn si OFia 1 i ty to provide evidence that the 
basic assumption of the item response models is met: 

A rough procedure i s to compute the latent roots of the tetrachori c 
item vfrtercorrelation matrix with estimated communal i ties placed in 
the diagonal. If (1) the first root is 1 arge compared to the second 
and (2 s ! the second root is not much' larger than any of the others, 
then the items are approximately unidimensional (p. 21). 

Thus^we have another factor analytic definition of dimensionality. 

In light of the brief discussion above, it seems reasonable to recon-. 
sider the rationale behind the research, being conducted for^ this 
symposium. Specifically,, three methods for analyzing multidimensional 
item response data are being compared v/i th respect to their abilities to 
recover the underlying dimensionality of some systematically constructed 
real test data, fleing real test data, their true dimensionality cannot be 
known with certainty even by the moderator or discussant. For example, 
the subtests from which the items were drawn may not have themselves been 
purely unidimensional, or the subtests may have been i ntercdrrel ated to 
some degree. Because the 'content areas measured by the items are unknown 
to the investigators, subject matter cannot, serve as a subjective guide 
for interpreting the results (often a self-fulfilling prophecy). 

However, these objections in no way negate the merit of the present 
research. They simply help to clarify, what is meant by the underlying 
dimensionality of the di chotomous item response d«ata being analyzed. A 
reasonable operational definition of dimensionality recovery might be 
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stated as the degree to which a technique is 'able to. classify accurately 
each of the items as belonging to the subtest from which it was drawn ori- 
ginally. 

Choice of Proximity Coefficients 

Typically, .the.data that serve. as input to MDS procedures are scored on 
'at. least the level of a so-called ordinal scale. Also, in the classical 
MDS situation, the.data are ar-anged. as a square, symmetric matrix con- 
taining the proximities of each variable With every other variable, simi- 
lar to an . i ntercorrel ati on matrix. Normally only the set of lower 
off-c.i agonal elements is used as input because the upper off-diagonal ele- 
ments are redundant and the proximi ties of variables with themselves along 
the diagonal are not of interest. However, with an array of dichotomous 

c data, " none of the above conditions hold: Because the items are 
scored as either 0 or 1, the data exi stj at the 1 evel of a so-called nominal 
scale, ^there being just two categories of responses: From an extrinsic 
point of view, order information is present in the binary response data in 
the sensa that achi evement v 1 evel has been mapped into two elements of an 
ordered set. The element 1 is "better than 11 the element 0 in that it 
denotes a higher level of achievement. But intrinsi.ca i ly , because there 
are only two elements in the set,, we may just as we 11 call it an unordered 
set: Certainly the assignment of the numbers (0,1) is arbitrary. More- 
over, measures of association between any given pair of items are 
insensitive to the ordering (1>0). So we have a situation in which the 
raw data do not contain the order information we require for MDS. 

i . _ . ... .... 

Another difficulty is that the binary data arranged, i r the rectangul ar 
matrix are more properly characterized as dominance data rather than prox- 
imity data. That is, the data values represent the dominance relations of 
persons versus items, persons versus persons, and items. versus itoms. 
Therefore, the data must be transformed -into a square matrix of proximi- 
ties that reflect the associations among the items. 

Fortunately, a large number of well known measures of association have 
been developed for the case of two nominal variables crossed in a 2 x 2 
table: Unfortunately, we seek proximity measures that are relatively 
unaffected by variations in item difficulty levels and item guessing lev- 
els if we are to avoid artifacts -in the MDS solutions. 

Only one study was found 'in the literature (Reckase, 1982) which 
addressed the choice of a proximity coefficient and its effect, on MDS ana- 
lyses of dichotomous test data. Also investigated in the study were the 
effects of such variable's as item di f f icul ty , di scr imi^nati on , and guess- 
ing levels and the factor structure of the item response data. Thirteen 
coefficients were studied using simulated 50 item data-sets (1,000 exami- 
nees) with one, two, three, or nine factors present, both with and without 
random guessing effects |n the data. A real, data-set with two factors, was 
also investigated: The coefficients included the phi,, phi/phi max, tetra- 
choric, agreement, approval, eta, kappa, koppa , Yule's Q, Yule's Y, 
Goodman and Krbskal's gamma, Kendall's tau B, and the Lijphart index. 

With the simulated data-s_ets, the membership of the items to each of 
the factors was known, so MDS was evaluated in terms of its ability to 
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classify correctly the items onto their proper factors. In all of the MDS 
runs, the solutions were restricted to two-dimensional Euclidean spaces. 
Stress values (Kruskal , 1964) reflecting the goodness of fit of the MD£ 
model to the data <vefe also observed: 

In general, the study showed that the MDS results were in large part's 
function of the particular coefficient used to measure item proximities: 
Also, coefficients were identified which were sensitive to the effects of 
item difficulty and guessing levels relative to those which were not so 
affected. In particular, Yule's Y coefficient (to be described later) 
performed quite well for the simulated da^a bet 3 i n classifying the items 
correctly. However, none of the coefficients were able to classify prop- 
er ly the „ real data i terns wi thou t prior kni_wl edge of the i r. content areas . 
In additidhi.it was found that, items with low discrimination or high dif- 
ficulty levels were so dissimilar from the other items that they tended to 
compress the locations of the remai n i ng i terns together , thereby, di stort- 
ing clear patterns that might ptherwi se have been revealed in the space. 
The presence of guessing in the data tended to reduce the sizes of the 
proximity coefficients among the items, further clouding the structure. 

A frequently recommended procedure to transform rectangul ar data i n to 
proximity data has been to calculate the set of" squared Euclidean dis- 
tancesfrom each variable to every other variable ( Kruskal & Wi sh , 1978 ,; 
Schiffman, Reynolds, & Young , 1981 ) . I n 'the case of dichotomous item 
response data, th : procedure amounts to summing the squared differences 
oetween two column vectors at a time, each of which contains only 0 1 s or 
1 1 s . If the patterns of response to the two items were identical, this 
distance would be equal to zero. If the patterns were exactly opposite, 
the distance would be equal to the sample, size, Also, it fs easy to see 
that this distance would be quite sensi ti ve (_i nf 1 ated) by large differ- 
ences in the item difficulty levels of the two items. 

Working within the framework of partial order analysis to address the 
issue of dimensi onal i ty of dichotomous data, Wise (1982) has proposed mak- 
ing modifications to the squared Euclidean distance. The sample size is 
used as a scaling constant and the'' item difficulty levels are taken into 
account by this new measure , cal 1 ed the Relative Proximity Index ( RPI ) . 
This index will be described in detail later. 

Proximity measures are commonly classified into two basic types, name- 
ly similarity data and dissimilarity data. Simi 1 ari ty data are defined 
such that, high positive values indicate that two variables are very simi- 
lar, while, low values indicate dissimilarity. Dissimilarity data are 
defined such that high positive values indicate a high degree of dissimi- 
larity, While low values indicate similarity.. The popular MDS computer 
routines require that the user specify the input data as belonging to 
either one type or the other. 

: This requirement presents a dilemma for proximity measures that range 
in value from +1.0 to -1.0, such as phi, tetrachoric, Yule^s Y, and many 
other well known coefficients of association. If we specify the data as 
being similarities, we force the procedure to view a ^zero relationship 
between two variables as having greater proximity or association than a 
negative relationship between the variables. Yet most would agree that a 
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high negative correlation, for instance, would provide as much an indi- 
cation of the strength of association between, two variables as the same 
correlation positive in sigh, and both would indicate an association 
stronger than zero; 

Fortunately, the item i n tercorrel ati ons . f or most test data, including 
achievement tests, tend to contain predominantly non-negative values* a 
phenomenon so pervasive as. to. have been canonized as the First. Law of 
Intelligence (Guttman, 1965). However, fcri situations where sizable. nega- 
tive values are encountered , even after reversing the values for variables 
scored in. opposite directions, the MDS model will have, serious probl ems in 
fitting the data, and_ the stress of the. solution wil 1 be high. A remedy 
for. the situation would be to use a coefficient of proximity having a low- 
er limit of 0.0 and a positive valued upper limit. 



METHOD 



p_rp x44aj^ ty--M e as-ures 

Three coefficients were chosen to transform the rectangular item 
response data into three separate square., symmetric matrices of proximi- 
ties for input to the MDS analyses. These coefficient: were the phi, 
Yule's Y, and the.. Re! at ive Proximity Index (RPI), which'are described 
below using the following 2x2 table. 

Item j 

o ' 1 



0 j a j b 

I I 
Item i I j 

I I 

1 | c | d 

I I 
i I 

a+c b+d N=a+b+c+d 



Note: a,b,c, and d are cell frequencies and N is the sample size. 



a+b 



| c+d 
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Phi Coefficient 



The phi coefficient is the well known special case of the Pearson prod- 
uct moment correlation coefficient between two dichotomous variables. 
Its formula is given by 

i. 

phi = (ad - be) / [(a + c)(b + d)(a + b)(c + dj] ? . (I) 

The phi coefficient was included as a proximity measure for two. mai h. rea- 
sons -r it. is known to be sensitive to variations in the respective diffi- 
culty levels of the two items being compared and, being so widely used, it 
was. to serve as a benchmark against which, the other two proximity measures 
could be con-pared. It was treated as similarity data. 

Yule's Y Coefficient 

Yule's Y, also known as the coefficient of colligation, is formulated 
specifically to be independent of the marginals of the 2x2 table. 
Indeed, the Reckase (1982) study found Yule's Y to be relatively unaf- 
fected by item difficulty variations. 

i_ 1 JL l. 

Yule's Y = [(ad) P - (bc) ? ]/[(ad) ? + (bcT ] (2) 

Yale's Y is jointly inonotonic with Yule's Q, is a transformation of the 
tetrachori c coefficient, and is a special case of Goodman and Kruskal's 
gamma. Because of. these properties, MDS . results woul d be expected to hp 
virtual ly - identical for all four coefficients. Thus.,. Yule's Y Was 
included because it represents a class of coefficients and because it per- 
formed very well with simulated data in the Reckase study. It was treated 
as similarity data also. 



RPI 

The RPI (Wise, 1982) is basically" the squared Euclidean distance 
between the responses to two dichotomous items corrected for sample size 
and item difficulty; 

RPI = (b+c)/N[(((+d)/N)(l-((b^d)/N))+((b+d)/N)(l-((c^d)/N))] (3) 

The sample size is used simply as a scaling constant, while the other 
terms in the denominator equal the expected squared Eucljdean distance for 
independent items with the same difficulty levels (marginals) as the val- 
ues, found in the 2x2 table. Small values, of the RPI (its lower bound is 
0.0) indicate close proximity between two items, so the RPI was treated as 
dissimilarity data. It was included because the squared Euclidean dis- 
tance is often recommended for the transformation to proximity data, 
because it does not take on negative val ues , and because d[ssimi 1 ari ty 
data are usually preferred to similarity data in MDS applications. 

MDS Computer Programs 

A wide variety of progrs-.s is available to perform MDS analyses. For 
an excellent review, see Schiffman, Reynolds, & Young (1982). Both ALSCAL 
(Takane, Young, & deLeeuw, 1977) and MINISSA (Roskam & Lingoes, 1970) were 



7 



8 



chosen for use in the present research. Through, proper choice of program- 
options, both . perform classical, nonmetric MDS V but there are several 
important differences. _ The ALSCAL program employs an iterative least 
squares monbtonic transformation between the .input data and their. corre- 
sponding Euclidean distances in space, while attempting, to minimize 
squared stress. The HINISSA program, uses an iterative rank image mono ton- 
ic . transformation procedure 4 minimizes Guttman's,. coefficient of 
alienation, and rotates the final sol uti on . space along principal axes, 
although numerous other options are available, 

Procedure 

The ALSCAL program was run separately for the three forms of proximity 
data, specifying a range of solutions from one dimension to stx dimensions 
for each. The objective was to obtain stress values as a function of 
increasing dimensionality for each of the three coefficients. Scree-like 
plots of stress versus number of dimensions were used to assist in the 
determination of the "true" dimensionality of the data. Comparisons of 
the plots were also made with published Monte Carlo, results from MDS ^ 
data-sets containing various levels of random error. In addition, rough 
comparisons of stress values were made, with the expected val ues of stress 
both for random data and for data that fit the model (MacCallum, 1981). 

The ALSCAL program also provides values for the squared correlation 
between the Eucl i dean di stances based, on the final solution and the corre- 
sponding disparities obtained during the monotonic transformation. 
Roughly speaking, the squared correl ati on indicates the proportion of 
variance in the ori gi nal i nput data that is accounted for by the MDS 
model. The R-squared values were also plotted as a function of. the number 
of dimensions in the solution. 

Taking into consideration the magnitude of the stress values and the 
R-squared values, as well as the shape of the plots and Monte Carlo guide- 
lines, tiie dimensionality of the optimal solution was determined. Once 
decided, this solution was pre-speci f i ed and submitted to MINISSA for rep- 
lication, purposes, and to obtain a rotation of the configuration along 
principal axes.. Guttman has found the .1 atter. procedure to facilitate 
interpretations in numerous empirical studies (1982). 

Being derived from proximity i nf ormati on , the locations of the Hems in 
the final spatial configuration provide information about the similari- 
ties of all the items to each other. Therefore, cluster analysi s of the 
item coordinates was performed to identify homogeneous groupings of items 
in the space. The goal was to classify ea,ch of the items into categories 
and/or regions of the space that could be surmised to represent subtest 
membership. The CLUSTER: procedure in the SAS computer package was used to 
perform the cluster analyses, using hierarchical clustering based on 
Euclidean distances among the items. 

Finally, in an attempt to .ascertain the effects of traditional item 
discrimination and item, difficulty levels. (from item analysi s) on the 
final spatial configuration, each was correlated with the set of coordi- 
nates locating each item on each of the idi mensi ons . 
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RESULTS 



Qi m ? h A] on al i ty 

The results from 'running the ALSCAL program for each of the three types 
of proxomity data are presented in Table 1 in terms of both stress and 
R-squared and are shown graphically in Figure i: As expected, with 
increasing dimensionality of the solution, the stress values decreased 
and the R-squared vajues increased. The RP I coefficients consistently, 
yielded the lowest stress and Highest R-squared values, while Yule's Y 
'always had the highest stress and lowest R-squared values for any partic- 
ular number of dimensions. However, the general patterns were the same 
for all three coefficients. 



Insert Table 1 about here. 



Using the equations given by MacCallum (1981), the expected values for 
stress using random data were computed to be .460, .354, .294, and .255 
for solutions with dimensionality equal to 2, 3, 4, and 5, respectively. 
Therefore, it appeared safe to conclude than the ALSCAL analyses in. the 
present study did not capitalize on chance unduly. Based on MacCal 1 urn 1 s 
tabled stress values for data that fjt the MDS model, the present dichoto- 
mous item response data would best be described as having moderate levels 
of random error rather than 1 ow random error . For the RPI solution in 
three dimensions , stress was about . 15 and R-squared was about .90, indi- 
cating a reasonably good fit of the MDS model to the data. 



Insert Figure 1 about here. 



The scree-1 i ke plots of stress and R-squared versus number of dimen- 
sions are i 1 1 ustrated in Figure 1 . Two fairly obvious "elbows" appeared 
in th'e plots, one at' the two dimensional solution and one at the three 
dimensional solution. For solutions' of four dimensions and higher, the 
rate of change in both stress and R-squared was constant, implying that 
the "true" elbow in the- plots was at the three dimensional solution.. 
Al so, the stress va'i ue for RPI in two dimensions was an unacceptably high* 
value of about .26. Therefore , the three dimensional sol ution was deemed 
to be optimal for the data. Based on the RPL input matrix, the ALSCAL pro- 
gram yielded the coordi nates that 1 ocated the 50 i terns in 
three-dimensional space. These, coordinates, along with the traditional 
item di f f iculty and item discrimination values from item analysis, are 
presented in Table 2. 



Insert Table 2 about here. 



The resul ts of running MINISSA. on the three coefficients, specifying 
solutions in three dimensions, paral leled the ALSCAL results. Kruskal's 
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stress val ues._were sliahtly lower than_for the ALSCALjresults ,. being .. 151 , 
.160, and.. .137 for phi", YuleJ.s Y, and RPI , respectively _ The. MINISSA pro- 
gram provides ..a measure of fit cal led ..the .coefficient, of alienation (K), 
sometimes preferred as a measure. of fit because, unlike stress,,, it. has 
both an upper and lower bound (stress has no upper .bound)". The K is 
defined as the square root of the quari.ti ty*. one minus the .coefficient of 
determi nati on ( R- squared) . Guttman, whil'e insisting that "no coefficient 
has anything to do with choosing dimensionality*" still suggests a "rule 
of thumb" '.hat K should be .15 or. less (1982). Because K-squared plus 
R-squared equal's 1. 0, a val ue* of .1*5 for K corresponds to a correlation 
coefficient .of about .99. The K for the RPI data in three dimensions was 
.144, again suggesting that reasonable fit was attained in three dimen- 
sions. , ; 

D s s c ripti ve Correl ations .> 

The correlations , between • the traditional item difficulty and item dis- 
crimination level s . from -item analysis and <the coordinates of the three 
dimensional MDS solutions are presented in Table 3. The main purpose of 
these analyses was to determine the degree to which the MDS configurations 
based on the phi,_Yule:'s Y, and the RPI proximities were affected by vari^ 
ations in item difficulty and. i_tem discrimination leyel.s. It. was expected 
that the item difficulty levels would correlate highly with one of the 
dimensions of. the MDS solution based' on phi coefficients and that the 
dimensions of the solutions based on Yule's Y and the RPI Would be less 
highly correlated with item difficulty. 



Insert Table 3 about here". 



Surprisingly, it was found that the first dimension of the RPI-based 
solution correlated quite highly (+.97) with item 'di ffieul ty. A similar 
result was' observed for the MDS solutions based on both phi and Yule 1 s Y, 
but not to .such an extent. Also', traditional item discrimination corre- 
lated moderately with 'several of the dimensions in each of the. sol uti on s . 
Thus, the attempt made in the present study to "Uti 1 ize proximity coeffi- 
cients that were relatively insensitive to the effects of item difficulty* 
(in particular) and item discrimination was not successful. 

CT- us te_r_ Anal y g-i-g 

The cluster analyses procedure used to i nterpret the MDS results and to 
classify items was restricted to the spatial configuration resulting from 
the RPI input matrix, mainly due to the fact that the MBS model had the 
best fit to the RPI proximity data. Also, the three-dimensional config- x 
urations were roughly the same for all three types of proximity data. 
However , because the proportion of variance in common between i itemjdi ff i- 
culty and the RPI coordinates on the first dimension was roughly ,94, the 
decision was made to perform the cluster analysis only on dimensions two 
and three, from the ALSCAL solution. In effect, this procedure removed the 
effect of i tern di ffieul ty ; from the cl usteri ng of items into homogeneous 
subsets. The plot of the items in two-dimensional space is shown in Fig- 
ure 2 . 
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Insert Figure 2 about here. 



Because Hierarchical clustering was performed, it was possible to fol- 
low the progress of the formation of item clusters on ; e step at a time. 
This progression of clustering provided a great deal more information than 
simply picking a stage of the hierarchical ci ustering_ results with some 
fixed number of clusters and attempting to c_iassi_fy_ the i tems_ by thair 
cluster membership at that stage. In particular; the patterns of interre- 
lations among the items' cquld_.be more accurately discerned by observing 
items that clustered together at one stage and' then either did or did not 
cluster with other items at subsequent' stages* Figure 3 presents . the 
results when 19 clusters remained, with previous clusterings indicated. 



F acet The or y / Regional Analysis ; 

Because of the distinctive pattern of clusters, the decision was made 
to use the Guttman facet theory/regional analysis method of interpreting 
the space . of the MDS solution ( see Levy ,1981 K for an excellent 
treatment). However, strict regional hypotheses expressed in the .form of 
mapping sentences could not ^e formulated due to the absence of knowledge 
about the content of the:itemsr Even so, extrapolating from the results 
of Guttman in the area of intelligence testi ng (1965.) , it was bel i eved • 
that two basic facets would partition, and describe the regional i ty of the 
space, one facet playing a polar role and the other playing a modular 



The polar facet was expected to divide the items into separate regions 
of the space, i n terms of subject matter or content area. Because no 
implied ordering is present, in the sense that no content area is better 
or ranked higher than another, this facet would simply divide the space 
using axes emanating from a pole or origin, much like slicing up a pie. 
Based on the cluster patterns, it appeared thatfour content areas (sub- 
tests of items) were present in the original i^tem response data. . Figure 4 
illustrates the polar axes and shows the items classified into eacfrcon- 
tent area : 



Achievement tests typically are constructed i n consideration of the 
fact that, test items measure performance at different cognitive levels. 
For example, "items are sometimes written deliberately to measure perform- 
ance iat the knowledge, comprehension, and appl i cation level s of Bloom 1 s 
taxonomy. In sileh^cases there 'is a fairly clear hierarchy that orders the 
items. When rank order information is present, the facet correspond!* ng to 



In sert Fi gure 3 about here. 



ro 1 e . 



Insert Figure 4 about here. 
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this aspect of the variables tends to play a modular role rather than a 
polar role, where the modular rol e_ can be representee! as a series of con- 
centric circles spreading out_ from the origin. The items that grouped 
together in the i nnermostc ircl e would be expected^to measure quite basic 
knowl edge or general jnformati on , perhaps i ncl udi rig fundamental rul es . 
Moving outward, successive circles would con tain 'i terns that were increas- 
ingly *speci fj'c and that might require applications of basic information, 
up to problem solving and decision making. 



Insert Figure 5 afaout here. 



The modular regions derived from. the present data are represented in 
Figure 5, with the polar facet sti.l T present in the space. Depending on 
the content of the items, these three circles might represent the know- 
ledge, comprehension, and application levels of cognitive performance, or 
perhaps some other taxonomi c combinations. At any rate, the general hier- 
archy of moving from general to speci fi c would be expected in terms of the 
actual i tern - content and the leveV of task performance required of the 
examinee. The results of the regional analysis are summarized in Table 4. 



Insert Table 4 about here. 



Finally, if Dimension I of the three-dimensional MDS solution were 
brought back into the analysis, we might say that a third facet of the 
item response data were present, namely. item difficulty. This facet would 
be expected to play. an axial role, being perpendicular to the other two 
facets on the circular base. Thus, we would think *of the MDS space as 
being in the basic shape of a cylinder, as shown in Figure 6. In other 
words, items could^be located at various heights above the two-dimensional 
surface previously described, depending on thei rdi fficuity levels. This 
shape has been described as a cylindrex (Levy, 1981): 



Insert Figure 6 about here. 
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DISCUSSION 



One . major objective, of the present research wa~s to attempt to find a 
proximity coefficient, for di chbtomoUs var'-abl es that was relatively unaf- 
fec\ed by variations. in item difficulty. The rationale was that itemdif^ 
ficulty would otherwise lead to the presence of an arti factual or nuisance 
dimension jn^the MDS results. Two coefficients that attempted to correct 
for item difficulty, Yule's Y and the RPI, were ^compared to phi, a coeffi- 
cient that is. known to be sensitive to 1 tern* di f f i-cul ty . The results 
showed 4 that item difficulty effects were present in' the MbS v . solutions 
regardless of the particular coefficient used. In fact, the RPI proximity 
measure, ostensibly including an adjustment to take item difficu'lty into 
account, produced coordinates on Dimension I that correl atedf almost per- 
fectly with i^em difficulty. 

These results suggest that the search for proximity measures" that are . 
insensitive to item difficulty may be mi sguided _ and f uti 1 es _ arid that a 
shift in thinking is required. Perhaps it should be acknowl edged that 
item difficulty is indeed a legitimate and reasonable > dimensi on along 
which items should be expected to vary.. Item difficulty may have to be 
regarded as one important aspect of di chotomous data that characterizes 
the sim_ilarities._..or dissimilarities of items to each other. In other 
words, from an_MDS perspective, item difficulty may be a real dimension 
and not an artifact. ^ 

The use of MDS to perform the classification of items into subtests was 
largely an exercise in subjectivity. Lingoes (1981) has argued that MDS 
analysis in the absence of. content is bound to be sterile, and some would 
claim that it is anti-scientific to perform a ''blind 11 analysis that is 
devoid of a theoretical or conceptual base: However, such methods as fac- 
tor analysis and MDS have frequently been criticized as being used i n ways 
to prove whatever it is that the researcher wishes to prove. Therefore, 
the present research paradigm could.be argued to comprise a relatively 
objective and fair test of three analytic procedures, a]beit an extremely 
frustrating one to the investigators. 

In the. absence of knowledge of item content, some popular methods of 
interpreting MDS results had to be ruled. out. These methods i ncl uded 
regression qr canonical correlation of external criteria on the MDS stimu- 
lus coordinates, as well as the use of the magnitudes and signs of the 
coordinates themselves to attempt to describe meaningful directions in 
the space. 

The Guttnian approach using facet theory and regional analysis to parti- 
tion the space could be applied, but not in standard fashi on ._. Thi s was 
because knowledge of the content of the variables was lacking. Therefore, 
no specific hypotheses could be formulated nor mapping sentences gener- 
ated: Despite these limitations,, sufficient results have. been published 
based on intelligence test data that the possibi 1 i ty existed to extrapo- 
late to unknown achievement test data. The degree to which the 
generalization was successful can be determined on ly wi th kn owl edge of the 
item contents and subject matter areas from which the original-'i terns were 
drawn . : 
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Table 1 

Stress and R-sqaared values from AbSCAb Analyses 



«- Stress R-squared 



Dimensions Phi Yule's Y RPI Phi Yule's Y RPI 



1 


.374 


.400 : 


.334 


. .669 


.626 


.724 


2 


.222 


.239 


. 195 


.809 


.780 


.353 


3 


. 163 


... 174 


. 146 


.861 


.836 


.891 


4 


. 136 


. 146 


.125 


.882 


.859 


.905 


5 


. 1 14 


. 123 


. 103 


.903 


.883 


.922 


6 


.096 


.104 


.089 


.920 


.904 


.933 



i 



is 



...Table 2 

Traditional . Item Difficulty and Di scri mi natioji Values 
and ALSCAL ^Coordinates based on RPI Coefficients 

> : T 



I tern 


Diff . 


D i s c r i m . 


Dim. I 


Dim 


. II 


Dim 


. Ill 


1 


.929 


/269 


1 


.828 


-1 


.443 


1 


.112 


2 


.863 


.250 


1 


. 528 


1 


.450 


1 


.309 


3 


.859 


.292 


, * 1 


. 633 


-0 


.810. 


1 


.059 


4 


.856 


. 424 


1 


.252 


0 


.373 . 


-0 


.281' 


5 


.832 


. 410 


1 


.151 


0 


.495 


-d 


, 102 


6 


.817 


.235 


o 


. 424 


-2 


.278 


0 


.818 


7 


. 79F 


. 450 


1 


.065 


-d 


.003 


0 


;469 


8 


. 793 


.487 


o 


. 860 


0 


;244 


-8 


;266 


9 


. 792 


. 479 


0 


999 


0 


;36i 


0 


.263 


10 


. 788 


. 309 


' 1 

1 


912 


0i 


/334 s 


0 


.234 


1 1 


. 772 


: 431 


± 


: 340 


GT: 372 


-8 


; 101 


12 


: 762 


.455 


n 

u 


£44 


h 
u 


. 4j / 


0 


.687 


13 


. 756 


: 421 


1 


. U 1 D 


-H 

y 


. cD4 


o 


. 162 


14 

-1 T 


. 755 


X 362 


1 


349 


n 
u 


. ooy- 




.463 


15 


748 


.410 


1 . 

J. 




n 

u , 




-0 


.498 


16 

1 u 


742 


?62 


n 

y 


491 


l 




o 


.432 


17 


733 

. Z_ «J «J 


. 558 


n 

VJ 


644 

• y zr 


n 

u , 


?n? 


-o 


. 118 


18 

1 o 


723 

• L. w J 


. 356. 


n 
u 


. lOU 


u 


QSQ 
. yC J 


-b. 


.945 


19 


721 


. 402 


n 

u 


• PPA 


n 

u , 


RfiQ 

. OU 3 


-0 , 


.814 


2H 
c. y 


71 4 


285 


1 

1 


1 fi? 


u , 


91 A 
. C 1 t 


-1 , 


. 542 


21 


7D2 


472 


n 


74D 


n 

u , 


ICC 
.10 0 


-0 , 


.414 


c c 


7nn 


. 507 


n 


1 9^ 


n 

u , 


.0/1 


0 , 


.084 


c o 


7nn 


398 


n 

u 




u , 


.300 


0 , 


; 362 


C H 


. y / o 


3fil 


h 

u 


7RR 


n 

u . 




-J ; 


.075 


?"c 

C J 


fi72 


Rl 9 


n 


?4R • 


Q. . 


1*+/ 


6 . 


, 209 




fi46 


4ft3 


U 


274 


u , 




-0 , 


.689 


27 


64R 




u 


. u *+y 


u , 




o, 


. 573 


2fi 


634 


429 


u 




n 


976 

. Z_ W 


o , 


. 593 


9Q 


627 


514 


n 

u 


H44 1 


0. 


.474 


o . 


. 340 


o U 


"R72 


297 


-n 

y 


S79 




,527 


-0 , 


.973 


34 

O I 


RR3 


RD4 


-n 

y 


27D 


0, 


.666 


-0 , 


. 192 


32 


561 


.219 


-n 


. JUO 


-2. 


,551 


0 , 


. 152 


33 


.540 


. 447 


-n 


3 1 R 


0. 


.118 


o , 


.281 


34 


523 


.412 


-n 

y 


. y / u 


0. 


,916 


o , 


.627 


3R 


51 5 


. 461 


-n 


642 


0. 


,441 


-0 , 


.321 


3fi 
o u 


. ^ w ^ 


. 494 


-n 

y 


RRn 

. J J u 


0. 


,124 


0 1; 


,225 


37 


.456 


. 267 


u 


R77 


-2. 


259 


-0; 


,266 


38 


. 445 


. 395 


-0 


. 914 


0. 


173 


-6. 


,668 


■J Z/ 


. 442 


. 405 


-0 


.420 


1. 


625 


~0; 


456 


40 


; 431 


. 429 


-0 


.489 


0. 


756 


0. 


,679 


4i 


X :373 


.301 


-i. 


.279 


1. 


,460 


-1. 


062 


42 


:373 


.356 


=i. 


.481 


0. 


548 


0. 


870 


43 


:344 


.353 


-l. 


.369 


0. 


792 


-o. 


,767 


44 


.328 


.282 


-2. 


.218 


0. 


,069 


0. 


,692 


45 


.294 


.327 


-1 , 


.631 


1 . 


332 


0. 


349 


46 


.271 


.207 


-2. 


.543 


-b. 


422 \ 


1. 


281 


47 


- .256 


.333 


-1 , 


.898 


0. 


070 


-0. 


341 


48 


.241 


.262 


-2. 


.151 


-0. 


852 


-1. 


113 


49 


.226 


.209 


-2. 


.389 


-d. 


558 


1. 


766 


,50 


.146 


.086 


-2. 


660 


-2. 


148 


-2. 


162 
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Table 3 



Correlations Among Traditional Item Parameters and 
the ALSCAL Coordinates of the MDS Solution Spaces 



Item. Item . * 

Variable Di ff icul ty Di scrimi nati on 



Dimension 


1 


(phi) 




-.77 


-.59 


Dimension 


II. 


(phi) 




.49 


-.27 


Dimension 


Ill 


(phi) 




.09 


.01 


Dimension 


i 


(Yule 1 


s Y) 


.70 


.61 


Dimension 


ii 


(Yule' 


s Y) 


— .48 


.26 


Dimension 


in 


(Yule' 


s Y) 


-.05 


.06 


Dimension 


i 


(RPI) 


0 


.97 


.40 


Dimension 


ii 


(RPI) 




.03 


:53 


Dimension 


in 


(RPI) 




. 16 


.02 



Table 4 

Items Classified into Regions of the MDS Space 



Level Subtest I Subtest II Subtest III Subtest IV 



4,5,8,11, 7,9,10,29, 13,23,25 

General 15,17,21, 33,36,44 

35,47 

14,19,22, 12,27,28, 1,3,16, 18,48 / 

Applied 24,26,31, 34,40,42 46 f 
38,39,43 

Specific 20,41 ' 2,45 6,32,37, 30,50 

49 
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Figure 2 



Ai;St'.A!i Coordinates for ,)IM. II vs. DIM. Ill Based on RPI Coefficients 
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DIMENSION II 



+2.0 



15 

38 26 



14 

43 

i l9 



39 



24 



41 



19 



Figure 3 



Hierarchical Cluster Analysis of the MDS Spatial Configuration 
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Figure 5 



Tlie Addition of the Modular Facet, Ordering Items from Genera! to Specific 
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I f IE ANALYSIS OF DICH6TCMQUS TEST DATA 
USING NON METRIC MULTIDIMENSIONAL SCALING 

William R, Koch 
The University of Texas at Austin 



ABSTRACT 



The technique of nonmetric multidimensional scaling (MDS) was applied to real 
item response data obtained from a multiplc-ctioice achievement test of unknown di- 
mensionality. The goal., was to classify the 50 items into the various subtests from which 
they were drawn originally, the latter being unknown to the investigator. Issues addressed 
in the research included dimensionality, choice of item proximity measures, and appro- 
priates* of the MDS model for analyzing dichotomous item response data. Three coef- 
ficients were chosen to form proximity matrices that reflected the associations of each 
item with each of the other items. These matrices, then served as input to both the 
ALSCAL and MINISSA computer programs for MDS analysis. A three dimensional 
solution was found to be optimal, based on stress, and coefficient of alienation 
values. Both cluster analysis and regional (facet theory) analysis of the spatial config- 
uration were used to mterpret the results. 



